Udacity - Self-Driving Car NanoDegree

Udacity Self-Driving Car Engineer Nanodegree

Project: Traffic Sign Recognition Classifier

This project aims to build a classifier that is able to classify traffic sign with advanced computer vision technique and deep neural net work model.

Method

A overview of the method for the project can be found here.

Data set

The classifier is trained, validated and tested with over hundreds of thousands of examples among 43 classes of traffic signs from the data base of German Traffic Signs Recognition Benchmark GTSRB.

References

More technical details can be found at Sermanet and LeCun, 2011 Ciresan et al., 2012


Step 0: Load The Data

In [5]:
## Download and upzip data set, only need to be done once
## download 
# curl -O https://d17h27t6h515a5.cloudfront.net/topher/2017/February/5898cd6f_traffic-signs-data/traffic-signs-data.zip

## unzip
# unzip traffic-signs-data.zip
In [6]:
## Import relavent libariries
import pickle
import numpy as np
import pandas as pd
import cv2
import tensorflow as tf
import time
import scipy.ndimage
import matplotlib.pyplot as plt
from tensorflow.contrib.layers import flatten
from sklearn.model_selection import train_test_split
from sklearn.utils import shuffle
%matplotlib inline
In [7]:
## Load pickled data, read training, validating and testing data
training_file = 'train.p'
validation_file = 'valid.p'
testing_file = 'test.p'

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(validation_file, mode='rb') as f:
    valid = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
X_train, y_train = train['features'], train['labels']
X_valid, y_valid = valid['features'], valid['labels']
X_test, y_test = test['features'], test['labels']

# Read signData
sign_file = 'signnames.csv'

with open(sign_file, mode = 'r') as f:
    signData = pd.read_csv(f)

Step 1: Dataset Summary & Exploration

The pickled data is a dictionary with 4 key/value pairs:

  • 'features' is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).
  • 'labels' is a 1D array containing the label/class id of the traffic sign. The file signnames.csv contains id -> name mappings for each id.
  • 'sizes' is a list containing tuples, (width, height) representing the original width and height the image.
  • 'coords' is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES

1. 1 Provide a Basic Summary of the Data Set Using Python, Numpy and/or Pandas

In [8]:
## Examine basics of data set

# Number of training examples
n_train = X_train.shape[0]

# Number of validation examples
n_validation = X_valid.shape[0]

# Number of testing examples.
n_test = X_test.shape[0]

# Shape of an traffic sign image
image_shape = X_train.shape[1:3]

# Number unique classes/labels there are in the dataset.
n_classes = signData.shape[0]

# number for class, counts for training data'
trainClass, trainIdx, trainCounts = np.unique(y_train, return_index = True, return_counts = True)
validClass, validCounts = np.unique(y_valid, return_counts = True)
testClass, testCounts = np.unique(y_test, return_counts = True)

def plotSignExample(trainClass,trainCounts,trainIdx):
    fig = plt.figure(figsize=(12,8))
    for idxClass, count, idxStart in zip(trainClass,trainCounts,trainIdx):
        idxRnd = np.random.randint(idxStart, idxStart + count, 1)
        ax = fig.add_subplot(5, 10, idxClass+1)
        plt.imshow(X_train[idxRnd].squeeze())
    plt.show()
    return fig
In [9]:
fig = plotSignExample(trainClass,trainCounts,trainIdx)
fig.savefig('examples/DataSetVisual.png')

print("Number of training examples =", n_train)
print("Number of validation examples =", n_validation)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes in sign system =", n_classes)
print("Number of classes in training data =", trainClass.shape[0])
print("Number of classes in validating data =", validClass.shape[0])
print("Number of classes in testing data =", testClass.shape[0])
print(signData['SignName'])
Number of training examples = 34799
Number of validation examples = 4410
Number of testing examples = 12630
Image data shape = (32, 32)
Number of classes in sign system = 43
Number of classes in training data = 43
Number of classes in validating data = 43
Number of classes in testing data = 43
0                                  Speed limit (20km/h)
1                                  Speed limit (30km/h)
2                                  Speed limit (50km/h)
3                                  Speed limit (60km/h)
4                                  Speed limit (70km/h)
5                                  Speed limit (80km/h)
6                           End of speed limit (80km/h)
7                                 Speed limit (100km/h)
8                                 Speed limit (120km/h)
9                                            No passing
10         No passing for vehicles over 3.5 metric tons
11                Right-of-way at the next intersection
12                                        Priority road
13                                                Yield
14                                                 Stop
15                                          No vehicles
16             Vehicles over 3.5 metric tons prohibited
17                                             No entry
18                                      General caution
19                          Dangerous curve to the left
20                         Dangerous curve to the right
21                                         Double curve
22                                           Bumpy road
23                                        Slippery road
24                            Road narrows on the right
25                                            Road work
26                                      Traffic signals
27                                          Pedestrians
28                                    Children crossing
29                                    Bicycles crossing
30                                   Beware of ice/snow
31                                Wild animals crossing
32                  End of all speed and passing limits
33                                     Turn right ahead
34                                      Turn left ahead
35                                           Ahead only
36                                 Go straight or right
37                                  Go straight or left
38                                           Keep right
39                                            Keep left
40                                 Roundabout mandatory
41                                    End of no passing
42    End of no passing by vehicles over 3.5 metric ...
Name: SignName, dtype: object

1. 2 Include an exploratory visualization of the dataset

Visualize the German Traffic Signs Dataset using the pickled file(s).

In [10]:
## Plot the distributions of classes in each data set
def plotDistOri(trainClass,validClass,testClass,trainCounts,validCounts,testCounts):
    fig = plt.figure(figsize=(15,5))
    barW = 0.3
    plt.bar(trainClass - barW, trainCounts, width = barW, align = 'center')
    plt.bar(validClass, validCounts, width = barW, align = 'center')
    plt.bar(testClass + barW, testCounts, width = barW, align = 'center')
    plt.xlabel('Class')
    plt.ylabel('Number of examples')
    plt.legend(['Train','Valid','Test'])
    plt.show()
    return fig

def plotDataOri(X,y,signData):
    Class, Idx, Counts = np.unique(y, return_index = True, return_counts = True)
    nDemo = 10
    for idxClass, count, idxStart in zip(Class,Counts,Idx):
        print('Class = ',idxClass,',',signData.SignName[idxClass], ',',count,' examples')
        fig = plt.figure(figsize=(nDemo,2))
        idxRnd = np.random.randint(idxStart, idxStart + count, nDemo)
        for j in range(1, nDemo):
            ax = fig.add_subplot(1, nDemo, j)
            ax.imshow(X[idxRnd[j]])
        plt.show()
In [11]:
## Plot individual examples in each class in training data
fig = plotDistOri(trainClass,validClass,testClass,trainCounts,validCounts,testCounts)
fig.savefig('examples/DataSetDistribution.png')
In [169]:
## Plot randomly selected examples in each class in training data
plotDataOri(X_train,y_train,signData)
Class =  0 , Speed limit (20km/h) , 180  examples
Class =  1 , Speed limit (30km/h) , 1980  examples
Class =  2 , Speed limit (50km/h) , 2010  examples
Class =  3 , Speed limit (60km/h) , 1260  examples
Class =  4 , Speed limit (70km/h) , 1770  examples
Class =  5 , Speed limit (80km/h) , 1650  examples
Class =  6 , End of speed limit (80km/h) , 360  examples
Class =  7 , Speed limit (100km/h) , 1290  examples
Class =  8 , Speed limit (120km/h) , 1260  examples
Class =  9 , No passing , 1320  examples
Class =  10 , No passing for vehicles over 3.5 metric tons , 1800  examples
Class =  11 , Right-of-way at the next intersection , 1170  examples
Class =  12 , Priority road , 1890  examples
Class =  13 , Yield , 1920  examples
Class =  14 , Stop , 690  examples
Class =  15 , No vehicles , 540  examples
Class =  16 , Vehicles over 3.5 metric tons prohibited , 360  examples
Class =  17 , No entry , 990  examples
Class =  18 , General caution , 1080  examples
Class =  19 , Dangerous curve to the left , 180  examples
Class =  20 , Dangerous curve to the right , 300  examples
Class =  21 , Double curve , 270  examples
Class =  22 , Bumpy road , 330  examples
Class =  23 , Slippery road , 450  examples
Class =  24 , Road narrows on the right , 240  examples
Class =  25 , Road work , 1350  examples
Class =  26 , Traffic signals , 540  examples
Class =  27 , Pedestrians , 210  examples
Class =  28 , Children crossing , 480  examples
Class =  29 , Bicycles crossing , 240  examples
Class =  30 , Beware of ice/snow , 390  examples
Class =  31 , Wild animals crossing , 690  examples
Class =  32 , End of all speed and passing limits , 210  examples
Class =  33 , Turn right ahead , 599  examples
Class =  34 , Turn left ahead , 360  examples
Class =  35 , Ahead only , 1080  examples
Class =  36 , Go straight or right , 330  examples
Class =  37 , Go straight or left , 180  examples
Class =  38 , Keep right , 1860  examples
Class =  39 , Keep left , 270  examples
Class =  40 , Roundabout mandatory , 300  examples
Class =  41 , End of no passing , 210  examples
Class =  42 , End of no passing by vehicles over 3.5 metric tons , 210  examples

Step 2: Design and Test a Model Architecture

Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.

With the LeNet-5 solution from the lecture, you should expect a validation set accuracy of about 0.89.

To meet specifications, the validation set accuracy will need to be at least 0.93.

There are various aspects to consider when thinking about this problem:

  • Neural network architecture (is the network over or underfitting?)
  • Play around preprocessing techniques (normalization, rgb to grayscale, etc)
  • Number of examples per label (some have more than others).
  • Generate fake data.

References: LeCun.

2. 1 Pre-process the Data Set

Pre-processing techniques used: Histogram equalization, grayscale, normalizatin

In [12]:
## Define parameters for pre-processing
opt = {'gamma': 1.5, 'gray': True}
In [13]:
## Define pipeline of pre-process data, followed with helper functions
def preProcess(X, y, opt):
    n = X.shape[0]
    dim = np.asarray(X.shape)
    gamma = opt['gamma']
    if opt['gray'] == True:
        dim[3] = 1
    X_pre = np.zeros(dim,dtype = np.float32)
    for i in range(0, n):
        xTemp = X[i]
        xTemp = gamma_correction(xTemp,gamma)
        xTemp = hist_equalization(xTemp)
        
        if opt['gray'] == True:
            xTemp = rgb2gray(xTemp)
        xTemp = scaleDown(xTemp)
        if opt['gray'] == True:
            X_pre[i] = xTemp[:,:,np.newaxis]
        else:
            X_pre[i] = xTemp        
    return X_pre, y

def scaleDown(img):
    return np.float32(img/255)
#     return np.float32((img - 128)/128)

def scaleUp(img):
    return np.uint8(img*255)
#     return np.uint8(img*128 + 128)

def gamma_correction(img,gamma):
    img = img/255
    img = cv2.pow(img,1/gamma)
    return np.uint8(img*255)

def hist_equalization(img):
    # hist equal in YUV space
    imgYUV = cv2.cvtColor(img, cv2.COLOR_RGB2YUV)
    # equalize the histogram of the Y channel
    imgYUV[:,:,0] = cv2.equalizeHist(imgYUV[:,:,0])
    # convert the YUV image back to RGB format
    imgOut = cv2.cvtColor(imgYUV, cv2.COLOR_YUV2RGB)
    return imgOut

def sharpen_img(img):
    """
    Applies the Gaussian Blur on original image using openCV library
    """
    gb = cv2.GaussianBlur(img, (3,3), 20)
    return cv2.addWeighted(img, 2, gb, -1.5, 0)

def rgb2gray(img):
    return cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)

def rgb2yuv(img):
    return cv2.cvtColor(img,cv2.COLOR_RGB2YUV)
In [14]:
## Examine the pre-processed data
def plotDataPre(X, X_pre, y, signData, opt):
    Class, Idx, Counts = np.unique(y, return_index = True, return_counts = True)
    nDemo = 10
    for idxClass, count, idxStart in zip(Class,Counts,Idx):
        print('Class = ',idxClass,',',signData.SignName[idxClass], ',',count,' examples')
        fig = plt.figure(figsize=(nDemo,2))
        idxRnd = np.random.randint(idxStart, idxStart + count, nDemo)
        for j in range(1, nDemo):
            ax = fig.add_subplot(2, nDemo, j)
            ax.title.set_text(str(idxRnd[j]))  
            ax.imshow(X[idxRnd[j]])
            ax = fig.add_subplot(2, nDemo, j + nDemo)
            if opt['gray'] == True:
                ax.imshow(X_pre[idxRnd[j]].squeeze(), cmap = 'gray')
            else:
                ax.imshow(X_pre[idxRnd[j]])
        plt.show()
        
def plotDemoPre(x,idx):
    fig = plt.figure(figsize=(8,2));
    plt.subplot(141); plt.imshow(x.squeeze());
    plt.title('Original ' + str(idx))
    xTemp = gamma_correction(x,2)
    plt.subplot(142); plt.imshow(xTemp)
    plt.title('Gamma correction')
    xTemp = hist_equalization(xTemp)
    plt.subplot(143); plt.imshow(xTemp)
    plt.title('Histogram equal')
    xTemp = rgb2gray(xTemp)
    plt.subplot(144); plt.imshow(xTemp, cmap = 'gray')
    plt.title('Grayscale')
    plt.show()
    return fig
In [16]:
## Pre-process traing, validating and testing data set
X_train_pre, y_train_pre = preProcess(X_train, y_train, opt)
X_valid_pre, y_valid_pre = preProcess(X_valid, y_valid, opt)
X_test_pre, y_test_pre = preProcess(X_test, y_test, opt)

X_train_pre.astype(np.float32);
X_valid_pre.astype(np.float32);
X_test_pre.astype(np.float32);
In [17]:
## Plot individual demo of pre-processing
# idxTest = [7774,7991,8109,8150]
idxTest = [8109,8150]

for i in range(len(idxTest)):
    fig = plotDemoPre(X_train[idxTest[i]],idxTest[i])
    title = 'examples/preProcessDemo' + str(idxTest[i]) +'.png'
    fig.savefig(title)
In [170]:
## Plot all pre-processed training data
plotDataPre(X_train, X_train_pre, y_train, signData, opt)
Class =  0 , Speed limit (20km/h) , 180  examples
Class =  1 , Speed limit (30km/h) , 1980  examples
Class =  2 , Speed limit (50km/h) , 2010  examples
Class =  3 , Speed limit (60km/h) , 1260  examples
Class =  4 , Speed limit (70km/h) , 1770  examples
Class =  5 , Speed limit (80km/h) , 1650  examples
Class =  6 , End of speed limit (80km/h) , 360  examples
Class =  7 , Speed limit (100km/h) , 1290  examples
Class =  8 , Speed limit (120km/h) , 1260  examples
Class =  9 , No passing , 1320  examples
Class =  10 , No passing for vehicles over 3.5 metric tons , 1800  examples
Class =  11 , Right-of-way at the next intersection , 1170  examples
Class =  12 , Priority road , 1890  examples
Class =  13 , Yield , 1920  examples
Class =  14 , Stop , 690  examples
Class =  15 , No vehicles , 540  examples
Class =  16 , Vehicles over 3.5 metric tons prohibited , 360  examples
Class =  17 , No entry , 990  examples
Class =  18 , General caution , 1080  examples
Class =  19 , Dangerous curve to the left , 180  examples
Class =  20 , Dangerous curve to the right , 300  examples
Class =  21 , Double curve , 270  examples
Class =  22 , Bumpy road , 330  examples
Class =  23 , Slippery road , 450  examples
Class =  24 , Road narrows on the right , 240  examples
Class =  25 , Road work , 1350  examples
Class =  26 , Traffic signals , 540  examples
Class =  27 , Pedestrians , 210  examples
Class =  28 , Children crossing , 480  examples
Class =  29 , Bicycles crossing , 240  examples
Class =  30 , Beware of ice/snow , 390  examples
Class =  31 , Wild animals crossing , 690  examples
Class =  32 , End of all speed and passing limits , 210  examples
Class =  33 , Turn right ahead , 599  examples
Class =  34 , Turn left ahead , 360  examples
Class =  35 , Ahead only , 1080  examples
Class =  36 , Go straight or right , 330  examples
Class =  37 , Go straight or left , 180  examples
Class =  38 , Keep right , 1860  examples
Class =  39 , Keep left , 270  examples
Class =  40 , Roundabout mandatory , 300  examples
Class =  41 , End of no passing , 210  examples
Class =  42 , End of no passing by vehicles over 3.5 metric tons , 210  examples
In [171]:
## Plot all pre-processed validation data
plotDataPre(X_valid, X_valid_pre, y_valid, signData, opt)
Class =  0 , Speed limit (20km/h) , 30  examples
Class =  1 , Speed limit (30km/h) , 240  examples
Class =  2 , Speed limit (50km/h) , 240  examples
Class =  3 , Speed limit (60km/h) , 150  examples
Class =  4 , Speed limit (70km/h) , 210  examples
Class =  5 , Speed limit (80km/h) , 210  examples
Class =  6 , End of speed limit (80km/h) , 60  examples
Class =  7 , Speed limit (100km/h) , 150  examples
Class =  8 , Speed limit (120km/h) , 150  examples
Class =  9 , No passing , 150  examples
Class =  10 , No passing for vehicles over 3.5 metric tons , 210  examples
Class =  11 , Right-of-way at the next intersection , 150  examples
Class =  12 , Priority road , 210  examples
Class =  13 , Yield , 240  examples
Class =  14 , Stop , 90  examples
Class =  15 , No vehicles , 90  examples
Class =  16 , Vehicles over 3.5 metric tons prohibited , 60  examples
Class =  17 , No entry , 120  examples
Class =  18 , General caution , 120  examples
Class =  19 , Dangerous curve to the left , 30  examples
Class =  20 , Dangerous curve to the right , 60  examples
Class =  21 , Double curve , 60  examples
Class =  22 , Bumpy road , 60  examples
Class =  23 , Slippery road , 60  examples
Class =  24 , Road narrows on the right , 30  examples
Class =  25 , Road work , 150  examples
Class =  26 , Traffic signals , 60  examples
Class =  27 , Pedestrians , 30  examples
Class =  28 , Children crossing , 60  examples
Class =  29 , Bicycles crossing , 30  examples
Class =  30 , Beware of ice/snow , 60  examples
Class =  31 , Wild animals crossing , 90  examples
Class =  32 , End of all speed and passing limits , 30  examples
Class =  33 , Turn right ahead , 90  examples
Class =  34 , Turn left ahead , 60  examples
Class =  35 , Ahead only , 120  examples
Class =  36 , Go straight or right , 60  examples
Class =  37 , Go straight or left , 30  examples
Class =  38 , Keep right , 210  examples
Class =  39 , Keep left , 30  examples
Class =  40 , Roundabout mandatory , 60  examples
Class =  41 , End of no passing , 30  examples
Class =  42 , End of no passing by vehicles over 3.5 metric tons , 30  examples

2.2 Augment data

More data are adding by rotating the signs to balance the distribution of class of signs

In [126]:
## Define parameters
optAug = {'angMax':5, 'angMin':-5}
In [127]:
## Define data augmentation method, followed by helper functions for augmenting data
def AugmentData(X,y,optAug):
    nClass, nIdx, nCounts = np.unique(y, return_index = True, return_counts = True)
    compRatio = (max(nCounts) / nCounts - 1)/2
    compRatio = compRatio.astype(np.int)
    addCounts = nCounts * compRatio
#     print(nCounts)
#     print(addCounts)

    dimX = np.asarray(X.shape)
    dimY = np.asarray(y.shape)

    for i in range(len(addCounts)):
        idxS = nIdx[i]
        idxE = nIdx[i] + nCounts[i]
            
        if addCounts[i] != 0:
            dimX[0] = addCounts[i]
            dimY[0] = addCounts[i]
            X_temp = np.zeros(dimX, dtype = np.float32)
            y_temp = np.zeros(dimY, dtype = np.uint8)
            idxAug = np.random.randint(idxS, idxE, size = addCounts[i])
            thetaAug = np.random.randint(optAug['angMin'], optAug['angMax'], size = addCounts[i])
            for j in range(idxAug.shape[0]):
                X_temp[j,:,:,:] = augRotate(X[idxAug[j]], thetaAug[j])
                y_temp[j] = y[idxAug[j]]
            X_aug_temp = np.append(X[idxS:idxE], X_temp, axis = 0)
            y_aug_temp = np.append(y[idxS:idxE], y_temp, axis = 0)

            if i == 0:
                X_aug = X_aug_temp
                y_aug = y_aug_temp
            else:
                X_aug = np.append(X_aug, X_aug_temp, axis = 0)
                y_aug = np.append(y_aug, y_aug_temp, axis = 0)
        else:
            if i == 0:
                X_aug = X[idxS:idxE]
                y_aug = y[idxS:idxE]
            else:
                X_aug = np.append(X_aug, X[idxS:idxE], axis = 0)
                y_aug = np.append(y_aug, y[idxS:idxE], axis = 0)

    print('Number of examples before augmentation: ' + str(X.shape[0]))
    print('Number of examples after augmentation: ' + str(X_aug.shape[0]))
    return X_aug, y_aug

def combineData(X1,X2,y1,y2):
    nClass1, nIdx1, nCounts1 = np.unique(y1, return_index = True, return_counts = True)
    nClass2, nIdx2, nCounts2 = np.unique(y2, return_index = True, return_counts = True)
    dimX = np.asarray(X1.shape)
    dimY = np.asarray(y1.shape)
#     print(nIdx1)
#     print(nIdx2)
#     print(nCounts1)
#     print(nCounts2)
    dimX[0] = X1.shape[0] + X2.shape[0]
    dimY[0] = y1.shape[0] + y2.shape[0]

    X = np.zeros(dimX, dtype = np.float32)
    y = np.zeros(dimY, dtype = np.uint8)
#     print(X1.shape)
#     print(X2.shape)
#     print(X.shape)
    idx = 0
    for i in range(len(nClass1)):
#         print(str(i) + 'th segment starts at' + str(idx))
        idxS1 = nIdx1[i]
        idxE1 = nIdx1[i] + nCounts1[i]
        idxS2 = nIdx2[i]
        idxE2 = nIdx2[i] + nCounts2[i]
        
        X[idx:idx+nCounts1[i]] = X1[idxS1:idxE1]
        y[idx:idx+nCounts1[i]] = y1[idxS1:idxE1]
#         print(str(y[idx]) + ' and ' + str(y[idx+nCounts1[i]-1]))
        idx = idx + nCounts1[i]
        
        X[idx:idx+nCounts2[i]] = X2[idxS2:idxE2]
        y[idx:idx+nCounts2[i]] = y2[idxS2:idxE2]
#         print(str(y[idx]) + ' and ' + str(y[idx+nCounts2[i]-1]))
        idx = idx + nCounts2[i]

#         print(str(i) + 'th segment finishes at' + str(idx - 1))
    return X, y

def splitData(X,y):
    X_train, X_valid, y_train, y_valid = train_test_split(X,y,test_size=0.2,random_state=888)
    return X_train, X_valid, y_train, y_valid

def augRotate(x, theta):
    xNew = scipy.ndimage.rotate(x, theta, reshape=False)
    return xNew

def augShift(x, shift):
    xNew = scipy.ndimage.shift(x, shift)
    return xNew
In [149]:
## Define plot function
def plotDistAug(y, y_aug):
    augClass, augIdx, augCounts = np.unique(y_aug, return_index = True, return_counts = True)
    oriClass, oriIdx, oriCounts = np.unique(y, return_index = True, return_counts = True)
    
    fig = plt.figure(figsize=(15,5))
    barW = 0.4
    plt.bar(oriClass - barW, oriCounts, width = barW, align = 'center')
    plt.bar(augClass, augCounts, width = barW, align = 'center')
    plt.xlabel('Class')
    plt.ylabel('Number of examples')
    plt.legend(['Original data set','Augmented data set'])
    plt.show()
    return fig
    
def plotDataAug(X,y,signData):
    Class, Idx, Counts = np.unique(y, return_index = True, return_counts = True)
    nDemo = 10
    for idxClass, count, idxStart in zip(Class,Counts,Idx):
        print('Class = ',idxClass,',',signData.SignName[idxClass], ',',count,' examples')
        fig = plt.figure(figsize=(nDemo,2))
        idxRnd = np.random.randint(idxStart, idxStart + count, nDemo)
        for j in range(1, nDemo):
            ax = fig.add_subplot(1, nDemo, j)
            ax.imshow(X[idxRnd[j]].squeeze(), cmap = 'gray')
        plt.show()

def plotDemoAug(x,idx):
    x = scaleUp(x)
    xAug = augRotate(x[0],10)   
    fig = plt.figure(figsize=(4,2));
    plt.subplot(121); plt.imshow(x.squeeze(),cmap = 'gray');
    plt.title('Pre-processed ' + str(idx))
    plt.subplot(122); plt.imshow(xAug.squeeze(),cmap = 'gray');
    plt.title('Augmented' + str(idx))
    return fig
In [145]:
## Generating augmented data
print('Combine original training and validation data set')
X_total_pre, y_total_pre = combineData(X_train_pre, X_valid_pre, y_train_pre, y_valid_pre)

print('Generating augmented data')
X_total_aug, y_total_aug = AugmentData(X_total_pre, y_total_pre, optAug)

print('Spliting augmented data into training and validation set')
X_train_aug,X_valid_aug,y_train_aug,y_valid_aug = splitData(X_total_aug,y_total_aug)
print('Dimension of augmented training data is ' + str(X_train_aug.shape))
print('Dimension of augmented validation data is ' + str(X_train_aug.shape))

# print('For validation set')
# X_valid_aug, y_valid_aug = AugmentData(X_valid_pre, y_valid_pre, optAug)
# print('For testing set')
# X_test_aug, y_test_aug = AugmentData(X_test_pre, y_test_pre, optAug)
Combine original training and validation data set
Generating augmented data
Number of examples before augmentation: 39209
Number of examples after augmentation: 58138
Spliting augmented data into training and validation set
(46510, 32, 32, 1)
(11628, 32, 32, 1)
In [146]:
## Plot demo of augmented data
idxAugDemo = [7991]

for i in range(len(idxAugDemo)):
    fig = plotDemoAug(X_train_pre[idxAugDemo],idxAugDemo)
    title = 'examples/AugDemo' + str(idxAugDemo[i]) +'.png'    
    plt.savefig(title)
In [150]:
## Plot augmented data distributions
fig = plotDistAug(y_total_pre, y_total_aug)
fig.savefig('examples/AugDataDist.png')
In [138]:
## Plot combined pre-processed training and validation data
plotDataAug(X_total_pre, y_total_pre, signData)
Class =  0 , Speed limit (20km/h) , 210  examples
Class =  1 , Speed limit (30km/h) , 2220  examples
Class =  2 , Speed limit (50km/h) , 2250  examples
Class =  3 , Speed limit (60km/h) , 1410  examples
Class =  4 , Speed limit (70km/h) , 1980  examples
Class =  5 , Speed limit (80km/h) , 1860  examples
Class =  6 , End of speed limit (80km/h) , 420  examples
Class =  7 , Speed limit (100km/h) , 1440  examples
Class =  8 , Speed limit (120km/h) , 1410  examples
Class =  9 , No passing , 1470  examples
Class =  10 , No passing for vehicles over 3.5 metric tons , 2010  examples
Class =  11 , Right-of-way at the next intersection , 1320  examples
Class =  12 , Priority road , 2100  examples
Class =  13 , Yield , 2160  examples
Class =  14 , Stop , 780  examples
Class =  15 , No vehicles , 630  examples
Class =  16 , Vehicles over 3.5 metric tons prohibited , 420  examples
Class =  17 , No entry , 1110  examples
Class =  18 , General caution , 1200  examples
Class =  19 , Dangerous curve to the left , 210  examples
Class =  20 , Dangerous curve to the right , 360  examples
Class =  21 , Double curve , 330  examples
Class =  22 , Bumpy road , 390  examples
Class =  23 , Slippery road , 510  examples
Class =  24 , Road narrows on the right , 270  examples
Class =  25 , Road work , 1500  examples
Class =  26 , Traffic signals , 600  examples
Class =  27 , Pedestrians , 240  examples
Class =  28 , Children crossing , 540  examples
Class =  29 , Bicycles crossing , 270  examples
Class =  30 , Beware of ice/snow , 450  examples
Class =  31 , Wild animals crossing , 780  examples
Class =  32 , End of all speed and passing limits , 240  examples
Class =  33 , Turn right ahead , 689  examples
Class =  34 , Turn left ahead , 420  examples
Class =  35 , Ahead only , 1200  examples
Class =  36 , Go straight or right , 390  examples
Class =  37 , Go straight or left , 210  examples
Class =  38 , Keep right , 2070  examples
Class =  39 , Keep left , 300  examples
Class =  40 , Roundabout mandatory , 360  examples
Class =  41 , End of no passing , 240  examples
Class =  42 , End of no passing by vehicles over 3.5 metric tons , 240  examples
In [139]:
## Plot augmented pre-processed total data set
plotDataAug(X_total_aug, y_total_aug, signData)
Class =  0 , Speed limit (20km/h) , 1050  examples
Class =  1 , Speed limit (30km/h) , 2220  examples
Class =  2 , Speed limit (50km/h) , 2250  examples
Class =  3 , Speed limit (60km/h) , 1410  examples
Class =  4 , Speed limit (70km/h) , 1980  examples
Class =  5 , Speed limit (80km/h) , 1860  examples
Class =  6 , End of speed limit (80km/h) , 1260  examples
Class =  7 , Speed limit (100km/h) , 1440  examples
Class =  8 , Speed limit (120km/h) , 1410  examples
Class =  9 , No passing , 1470  examples
Class =  10 , No passing for vehicles over 3.5 metric tons , 2010  examples
Class =  11 , Right-of-way at the next intersection , 1320  examples
Class =  12 , Priority road , 2100  examples
Class =  13 , Yield , 2160  examples
Class =  14 , Stop , 780  examples
Class =  15 , No vehicles , 1260  examples
Class =  16 , Vehicles over 3.5 metric tons prohibited , 1260  examples
Class =  17 , No entry , 1110  examples
Class =  18 , General caution , 1200  examples
Class =  19 , Dangerous curve to the left , 1050  examples
Class =  20 , Dangerous curve to the right , 1080  examples
Class =  21 , Double curve , 990  examples
Class =  22 , Bumpy road , 1170  examples
Class =  23 , Slippery road , 1020  examples
Class =  24 , Road narrows on the right , 1080  examples
Class =  25 , Road work , 1500  examples
Class =  26 , Traffic signals , 1200  examples
Class =  27 , Pedestrians , 1200  examples
Class =  28 , Children crossing , 1080  examples
Class =  29 , Bicycles crossing , 1080  examples
Class =  30 , Beware of ice/snow , 1350  examples
Class =  31 , Wild animals crossing , 780  examples
Class =  32 , End of all speed and passing limits , 1200  examples
Class =  33 , Turn right ahead , 1378  examples
Class =  34 , Turn left ahead , 1260  examples
Class =  35 , Ahead only , 1200  examples
Class =  36 , Go straight or right , 1170  examples
Class =  37 , Go straight or left , 1050  examples
Class =  38 , Keep right , 2070  examples
Class =  39 , Keep left , 1200  examples
Class =  40 , Roundabout mandatory , 1080  examples
Class =  41 , End of no passing , 1200  examples
Class =  42 , End of no passing by vehicles over 3.5 metric tons , 1200  examples
In [ ]:
## Plot augmented training data
plotDataAug(X_train_aug, y_train_aug, signData)
In [ ]:
## Plot augmented validation data
plotDataAug(X_valid_aug, y_valid_aug, signData)

2. 2 Model Architecture

In [151]:
## Define model parameters
optMod = {'epochs': 50, 'batchSize': 128, 'learnRate':0.0001,
         'keepRate':0.8, 'dataChoice': 'aug', 
          'initMu': 0, 'initStd': 0.1}
In [152]:
## Define each individual layer
def convLayer(x, fSize, stride, inDepth, outDepth, optMod):
    conW = tf.Variable(tf.truncated_normal(shape = (fSize, fSize, inDepth, outDepth), 
                                            mean = optMod['initMu'], stddev = optMod['initStd']))
    conB = tf.Variable(tf.zeros(outDepth))
    con = tf.nn.conv2d(x, conW, strides = [stride, stride, stride, stride],
                      padding = 'VALID') + conB
    con = tf.nn.relu(con)
    return con

def maxPool(x, fSize):
    return tf.nn.max_pool(x, ksize = [1, fSize, fSize, 1], strides = [1, fSize, fSize, 1],
                         padding = 'VALID')

def fullCon(x, inDim, outDim, optMod):
    W = tf.Variable(tf.truncated_normal(shape = (inDim, outDim), mean = optMod['initMu'], 
                                       stddev = optMod['initStd']))
    B = tf.Variable(tf.zeros(outDim))
    y = tf.matmul(x, W) + B
    y = tf.nn.relu(y)
    return y
In [153]:
## Construct full model
def fullModel(x, optMod):
    # input dimension = [32, 32, 1]    
    convL1 = convLayer(x, 5, 1, 1, 70, optMod)
    # output dimension = [28, 28, 70]
    
    maxPool1 = maxPool(convL1, 2)
    # output dimension = [14, 14, 70]
    
    convL2 = convLayer(maxPool1, 5, 1, 70, 140, optMod)
    # output dimension = [10, 10, 140]
    
    maxPool2 = maxPool(convL2, 2)
    # output dimension = [5, 5, 140]
    
    convL3 = convLayer(maxPool2, 3, 1, 140, 210, optMod)
    # output dimension = [3, 3, 210]
    
#     maxPool3 = maxPool(convL3, 2)
    # output dimension = [, , 210]
    
    fCon1 = flatten(convL3)
    fCon1 = tf.nn.dropout(fCon1, optMod['keepRate'])
    # output dimension = 3*3*210
    
    fCon2 = fullCon(fCon1, 3*3*210, 300, optMod)
    # output dimension = 300
        
    logits = fullCon(fCon2, 300, 43, optMod)
    # output dimension = 43
    
    return logits

2.4 Train, Validate and Test the Model

A validation set can be used to assess how well the model is performing. A low accuracy on the training and validation sets imply underfitting. A high accuracy on the training set but low accuracy on the validation set implies overfitting.

In [155]:
## Define training & validation data
if optMod['dataChoice'] == 'aug':
    X_train_mod = X_train_aug
    y_train_mod = y_train_aug
    X_valid_mod = X_valid_aug
    y_valid_mod = y_valid_aug
    print('The augmented training and validation data are used')
elif optMod['dataChoice'] == 'pre':
    X_train_mod = X_train_pre
    y_train_mod = y_train_pre
    X_valid_mod = X_valid_pre
    y_valid_mod = y_valid_pre
    print('The original training and validation data are used')
elif optMod['dataChoice'] == 'hyb':
    X_train_mod = X_train_aug
    y_train_mod = y_train_aug
    X_valid_mod = X_valid_pre
    y_valid_mod = y_valid_pre
    print('The hybrid training and validation data are used')
The augmented training and validation data are used
In [156]:
## Set up tensor flow training session
# Set up place holder for input data
x = tf.placeholder(tf.float32, (None, 32, 32, 1))
y = tf.placeholder(tf.uint8, (None))
one_hot_y = tf.one_hot(y, n_classes)
keep_prob = tf.placeholder(tf.float32)

logits = fullModel(x, optMod)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=one_hot_y, logits=logits)
loss_operation = tf.reduce_mean(cross_entropy)

optimizer = tf.train.AdamOptimizer(learning_rate = optMod['learnRate'])
training_operation = optimizer.minimize(loss_operation)
In [157]:
## Set up tensor flow session to evaluate accuracy and loss
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
saver = tf.train.Saver()

def evaluate(X_data, y_data, keepRate):
    num_examples = X_data.shape[0]
    total_accuracy = 0
    total_loss = 0
#     sess = tf.get_default_session()
    for offset in range(0, num_examples, optMod['batchSize']):
        batch_x, batch_y = X_data[offset:offset+optMod['batchSize']], y_data[offset:offset+optMod['batchSize']]
        [accuracy, loss] = sess.run([accuracy_operation, loss_operation], 
                                    feed_dict={x: batch_x, y: batch_y, keep_prob: keepRate})
        total_accuracy += (accuracy * len(batch_x))
        total_loss += (loss * len(batch_x))
    return (total_accuracy / num_examples, total_loss / num_examples)
In [158]:
# Plots history of learning curves for a specific model.
def plot_learning_curves(train_loss, valid_loss, train_acc, valid_acc):
    """
    Plots learning curves (loss and accuracy on both training and validation sets).
    """
    epoch = range(1, int(train_loss.shape[0])+1)
    fig = plt.figure(figsize = (10, 4))
    axis = fig.add_subplot(1, 2, 1)
    axis.plot(epoch,train_acc*100)
    axis.plot(epoch,valid_acc*100)
    axis.legend(['train','valid'])

    plt.grid()
    plt.xlabel("epoch")
    plt.ylabel("accuracy")
    plt.ylim(50, 110)

    axis = fig.add_subplot(1, 2, 2)
    axis.plot(epoch,train_loss)
    axis.plot(epoch,valid_loss)
    axis.legend(['train','valid'])

    plt.grid()
    plt.xlabel("epoch")
    plt.ylabel("loss")
    plt.ylim(0.001, 1) 
    return fig
In [159]:
## Train model and compute training and cross-validation error
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = X_train_mod.shape[0]
    
    train_loss_history = np.empty([0], dtype = np.float32)
    train_accuracy_history = np.empty([0], dtype = np.float32)
    valid_loss_history = np.empty([0], dtype = np.float32)
    valid_accuracy_history = np.empty([0], dtype = np.float32)
    
    print("Training...")
    print()
    for i in range(optMod['epochs']):
        timeStart = time.time()
        X_train_rand, y_train_rand = shuffle(X_train_mod, y_train_mod)

        for offset in range(0, num_examples, optMod['batchSize']):
            end = offset + optMod['batchSize']
            batch_x, batch_y = X_train_rand[offset:end], y_train_rand[offset:end]
            sess.run(training_operation, feed_dict={x: batch_x, y: batch_y, keep_prob: optMod['keepRate']})
        
        [train_accuracy, train_loss] = evaluate(X_train_mod, y_train_mod, 1)
        [valid_accuracy, valid_loss] = evaluate(X_valid_mod, y_valid_mod, 1)

        valid_loss_history = np.append(valid_loss_history, [valid_loss])
        valid_accuracy_history = np.append(valid_accuracy_history, [valid_accuracy])
        train_loss_history = np.append(train_loss_history, [train_loss])
        train_accuracy_history = np.append(train_accuracy_history, [train_accuracy])
        
        timeElapsed = time.time() - timeStart
        print("EPOCH %.0f, time used = %.1f sec" % (i+1, timeElapsed))
        print("Training accuracy = %.1f%%, training loss = %.3f" % (100*train_accuracy, train_loss))
        print("Validation accuracy = %.1f%%, validation loss = %.3f" % (100*valid_accuracy, valid_loss))
        print()
        
    saver.save(sess, './TrafficSignModelV7')
    print("Model saved")
Training...

EPOCH 1, time used = 25.7 sec
Training accuracy = 62.9%, training loss = 1.516
Validation accuracy = 62.4%, validation loss = 1.532

EPOCH 2, time used = 25.2 sec
Training accuracy = 78.7%, training loss = 0.820
Validation accuracy = 77.8%, validation loss = 0.854

EPOCH 3, time used = 25.2 sec
Training accuracy = 85.7%, training loss = 0.556
Validation accuracy = 84.5%, validation loss = 0.594

EPOCH 4, time used = 25.2 sec
Training accuracy = 90.2%, training loss = 0.367
Validation accuracy = 89.3%, validation loss = 0.405

EPOCH 5, time used = 25.2 sec
Training accuracy = 92.0%, training loss = 0.309
Validation accuracy = 91.3%, validation loss = 0.335

EPOCH 6, time used = 25.2 sec
Training accuracy = 93.0%, training loss = 0.266
Validation accuracy = 92.1%, validation loss = 0.294

EPOCH 7, time used = 25.2 sec
Training accuracy = 94.0%, training loss = 0.229
Validation accuracy = 92.8%, validation loss = 0.266

EPOCH 8, time used = 25.3 sec
Training accuracy = 94.6%, training loss = 0.209
Validation accuracy = 93.8%, validation loss = 0.237

EPOCH 9, time used = 25.2 sec
Training accuracy = 94.8%, training loss = 0.199
Validation accuracy = 93.9%, validation loss = 0.232

EPOCH 10, time used = 25.2 sec
Training accuracy = 95.2%, training loss = 0.188
Validation accuracy = 94.1%, validation loss = 0.220

EPOCH 11, time used = 25.2 sec
Training accuracy = 95.0%, training loss = 0.186
Validation accuracy = 94.1%, validation loss = 0.218

EPOCH 12, time used = 25.3 sec
Training accuracy = 95.4%, training loss = 0.175
Validation accuracy = 94.3%, validation loss = 0.210

EPOCH 13, time used = 25.3 sec
Training accuracy = 95.6%, training loss = 0.168
Validation accuracy = 94.7%, validation loss = 0.198

EPOCH 14, time used = 25.2 sec
Training accuracy = 95.9%, training loss = 0.159
Validation accuracy = 94.8%, validation loss = 0.193

EPOCH 15, time used = 25.2 sec
Training accuracy = 95.7%, training loss = 0.160
Validation accuracy = 94.9%, validation loss = 0.192

EPOCH 16, time used = 25.2 sec
Training accuracy = 96.0%, training loss = 0.153
Validation accuracy = 95.2%, validation loss = 0.181

EPOCH 17, time used = 25.2 sec
Training accuracy = 96.0%, training loss = 0.153
Validation accuracy = 95.1%, validation loss = 0.184

EPOCH 18, time used = 25.2 sec
Training accuracy = 96.1%, training loss = 0.149
Validation accuracy = 95.2%, validation loss = 0.181

EPOCH 19, time used = 25.2 sec
Training accuracy = 96.1%, training loss = 0.148
Validation accuracy = 95.3%, validation loss = 0.177

EPOCH 20, time used = 25.2 sec
Training accuracy = 96.2%, training loss = 0.146
Validation accuracy = 95.3%, validation loss = 0.179

EPOCH 21, time used = 25.2 sec
Training accuracy = 96.2%, training loss = 0.143
Validation accuracy = 95.4%, validation loss = 0.176

EPOCH 22, time used = 25.2 sec
Training accuracy = 96.3%, training loss = 0.143
Validation accuracy = 95.3%, validation loss = 0.174

EPOCH 23, time used = 25.2 sec
Training accuracy = 96.2%, training loss = 0.143
Validation accuracy = 95.4%, validation loss = 0.173

EPOCH 24, time used = 25.2 sec
Training accuracy = 96.2%, training loss = 0.143
Validation accuracy = 95.3%, validation loss = 0.176

EPOCH 25, time used = 25.4 sec
Training accuracy = 96.3%, training loss = 0.139
Validation accuracy = 95.4%, validation loss = 0.171

EPOCH 26, time used = 25.2 sec
Training accuracy = 96.3%, training loss = 0.141
Validation accuracy = 95.5%, validation loss = 0.168

EPOCH 27, time used = 25.3 sec
Training accuracy = 96.2%, training loss = 0.143
Validation accuracy = 95.4%, validation loss = 0.174

EPOCH 28, time used = 25.2 sec
Training accuracy = 96.3%, training loss = 0.140
Validation accuracy = 95.6%, validation loss = 0.166

EPOCH 29, time used = 25.2 sec
Training accuracy = 96.3%, training loss = 0.142
Validation accuracy = 95.4%, validation loss = 0.172

EPOCH 30, time used = 25.2 sec
Training accuracy = 96.4%, training loss = 0.137
Validation accuracy = 95.5%, validation loss = 0.169

EPOCH 31, time used = 25.2 sec
Training accuracy = 96.4%, training loss = 0.135
Validation accuracy = 95.7%, validation loss = 0.164

EPOCH 32, time used = 25.2 sec
Training accuracy = 96.4%, training loss = 0.137
Validation accuracy = 95.6%, validation loss = 0.170

EPOCH 33, time used = 25.2 sec
Training accuracy = 96.4%, training loss = 0.135
Validation accuracy = 95.6%, validation loss = 0.165

EPOCH 34, time used = 25.2 sec
Training accuracy = 96.4%, training loss = 0.137
Validation accuracy = 95.6%, validation loss = 0.170

EPOCH 35, time used = 25.2 sec
Training accuracy = 96.4%, training loss = 0.137
Validation accuracy = 95.6%, validation loss = 0.165

EPOCH 36, time used = 25.2 sec
Training accuracy = 96.4%, training loss = 0.137
Validation accuracy = 95.7%, validation loss = 0.162

EPOCH 37, time used = 25.3 sec
Training accuracy = 96.4%, training loss = 0.135
Validation accuracy = 95.7%, validation loss = 0.164

EPOCH 38, time used = 25.3 sec
Training accuracy = 96.4%, training loss = 0.135
Validation accuracy = 95.7%, validation loss = 0.162

EPOCH 39, time used = 25.2 sec
Training accuracy = 96.4%, training loss = 0.138
Validation accuracy = 95.6%, validation loss = 0.165

EPOCH 40, time used = 25.2 sec
Training accuracy = 96.5%, training loss = 0.134
Validation accuracy = 95.8%, validation loss = 0.160

EPOCH 41, time used = 25.2 sec
Training accuracy = 96.5%, training loss = 0.134
Validation accuracy = 95.9%, validation loss = 0.155

EPOCH 42, time used = 25.2 sec
Training accuracy = 96.5%, training loss = 0.133
Validation accuracy = 95.9%, validation loss = 0.155

EPOCH 43, time used = 25.3 sec
Training accuracy = 96.4%, training loss = 0.135
Validation accuracy = 95.9%, validation loss = 0.161

EPOCH 44, time used = 25.2 sec
Training accuracy = 96.4%, training loss = 0.135
Validation accuracy = 95.8%, validation loss = 0.163

EPOCH 45, time used = 25.2 sec
Training accuracy = 96.4%, training loss = 0.134
Validation accuracy = 95.8%, validation loss = 0.158

EPOCH 46, time used = 25.2 sec
Training accuracy = 96.4%, training loss = 0.134
Validation accuracy = 95.9%, validation loss = 0.159

EPOCH 47, time used = 25.2 sec
Training accuracy = 96.4%, training loss = 0.134
Validation accuracy = 95.8%, validation loss = 0.162

EPOCH 48, time used = 25.2 sec
Training accuracy = 96.4%, training loss = 0.136
Validation accuracy = 95.8%, validation loss = 0.161

EPOCH 49, time used = 25.2 sec
Training accuracy = 96.5%, training loss = 0.134
Validation accuracy = 95.8%, validation loss = 0.158

EPOCH 50, time used = 25.4 sec
Training accuracy = 96.4%, training loss = 0.135
Validation accuracy = 95.8%, validation loss = 0.159

Model saved
In [160]:
## Monitoring training and validation accuracy
fig = plot_learning_curves(train_loss_history, valid_loss_history, train_accuracy_history, valid_accuracy_history)
fig.savefig('examples/TrainingHistory.png')
In [161]:
with tf.Session() as sess:
#     new_saver = tf.train.import_meta_graph('TrafficSignModel.meta')
#     new_saver.restore(sess, tf.train.latest_checkpoint('.'))    
    saver.restore(sess,tf.train.latest_checkpoint('.'))
    [test_accuracy, test_loss] = evaluate(X_test_pre, y_test_pre, 1.0)
    print("Testing accuracy = %.1f%%, testing loss = %.3f" % (100*test_accuracy, test_loss))
Testing accuracy = 92.4%, testing loss = 0.372

Step 3: Test a Model on New Images

To give yourself more insight into how your model is working, download at least five pictures of German traffic signs from the web and use your model to predict the traffic sign type.

You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.

Load and Output the Images

In [162]:
## Load the images and plot them here.
import os
import PIL
from PIL import Image
In [163]:
def fetchNewData(lists):
    print(lists)
    i = 0
    X = np.empty([0, 32, 32, 3], dtype = np.uint8)
    for list in lists:
        image = Image.open('newTest/' + list)
        image = image.resize((32,32))
        xTemp = np.asarray(image, dtype=np.uint8)
        xTemp = xTemp[np.newaxis]
        X = np.append(X, xTemp, axis = 0)
        i = i + 1
    return X

def plotDataTest(X_new,y_new):
    i = 0
    fig = plt.figure(figsize=(20,3))
    for list in lists:
        image = Image.open('newTest/' + list)
        plt.subplot(1,10,i+1)
        image = image.resize((32,32))
        plt.imshow(image)
        xTemp = np.asarray(image, dtype=np.uint8)
        plt.title('Class ' + str(y_new[i]))
        xTemp = xTemp[np.newaxis]
        X_new = np.append(X_new, xTemp, axis = 0)
        i = i + 1
    return fig
In [164]:
lists = os.listdir("newTest/")
# y_new = np.asarray([16,1,33,11,38,18,12,25,35], dtype=np.uint8)
y_new = np.asarray([12,1,25,18,11,33,35,38,16], dtype=np.uint8)
X_new = fetchNewData(lists)

fig = plotDataTest(X_new,y_new)
fig.savefig('examples/NewTestData.png')
['00007.ppm', '00001.ppm', '00008.ppm', '00006.ppm', '00004.ppm', '00003.ppm', '00009.ppm', '00005.ppm', '00000.ppm']

Predict the Sign Type for New Images and Analyze Performance

In [165]:
## Run the predictions here and use the model to output the prediction for each image.
X_new_pre, y_new_pre = preProcess(X_new, y_new, opt)
In [166]:
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))
    [testNew_accuracy, testNew_loss] = evaluate(X_new_pre, y_new_pre, 1.0)
    print("New festing accuracy = %.1f%%, testing loss = %.3f" % (100*testNew_accuracy, testNew_loss))
New festing accuracy = 100.0%, testing loss = 0.000

Output Top 5 Softmax Probabilities For Each Image Found on the Web

For each of the new images, print out the model's softmax probabilities to show the certainty of the model's predictions (limit the output to the top 5 probabilities for each image). tf.nn.top_k could prove helpful here.

In [167]:
## Define helper function to print out the top ten softmax probabilities for the new test images
def plotDemoTopK(X, y, probTopK, idxTopK, nCase):   
    fig = plt.figure(figsize=(13,15))
    fig.subplots_adjust(hspace= 0, wspace = 1)

    for i in range(nCase):
        ax = fig.add_subplot(3,3,i+1)
        ax.imshow(X[i])
        label = topKLabel(y[i], probTopK[i],idxTopK[i])
        plt.xlabel(label,fontsize = 12)
        plt.show
    return fig

def topKPredict(X, k_top):
    sess = tf.get_default_session()
    probMatrix = sess.run(tf.nn.softmax(logits), feed_dict = {x: X, keep_prob: 1.0})
    topK = tf.nn.top_k(probMatrix, k = k_top)
    return sess.run(topK)

def topKLabel(idxCorr,prob, idx):
    s1 = str('Correct label is ' + signData['SignName'][idxCorr] + '\n')
    for j in range(len(prob)):
        s2 = (str(round(prob[j]*100,4)) + ' % being ' + signData['SignName'][idx[j]] + '\n')
        s1 = s1+s2
    return s1
In [168]:
nTopClass = 5

with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))
    topK = topKPredict(X_new_pre, nTopClass)
    
    fig = plotDemoTopK(X_new,y_new,topK.values, topK.indices, len(y_new))
    fig.savefig('examples/TopKDemo.png')

Project Writeup

Once you have completed the code implementation, document your results in a project writeup using this template as a guide. The writeup can be in a markdown or pdf file.

Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.


Step 4 (Optional): Visualize the Neural Network's State with Test Images

This Section is not required to complete but acts as an additional excersise for understaning the output of a neural network's weights. While neural networks can be a great learning device they are often referred to as a black box. We can understand what the weights of a neural network look like better by plotting their feature maps. After successfully training your neural network you can see what it's feature maps look like by plotting the output of the network's weight layers in response to a test stimuli image. From these plotted feature maps, it's possible to see what characteristics of an image the network finds interesting. For a sign, maybe the inner network feature maps react with high activation to the sign's boundary outline or to the contrast in the sign's painted symbol.

Provided for you below is the function code that allows you to get the visualization output of any tensorflow weight layer you want. The inputs to the function should be a stimuli image, one used during training or a new one you provided, and then the tensorflow variable name that represents the layer's state during the training process, for instance if you wanted to see what the LeNet lab's feature maps looked like for it's second convolutional layer you could enter conv2 as the tf_activation variable.

For an example of what feature map outputs look like, check out NVIDIA's results in their paper End-to-End Deep Learning for Self-Driving Cars in the section Visualization of internal CNN State. NVIDIA was able to show that their network's inner weights had high activations to road boundary lines by comparing feature maps from an image with a clear path to one without. Try experimenting with a similar test to show that your trained network's weights are looking for interesting features, whether it's looking at differences in feature maps from images with or without a sign, or even what feature maps look like in a trained network vs a completely untrained one on the same sign image.

In [ ]:
### Visualize your network's feature maps here.
### Feel free to use as many code cells as needed.

# image_input: the test image being fed into the network to produce the feature maps
# tf_activation: should be a tf variable name used during your training procedure that represents the calculated state of a specific weight layer
# activation_min/max: can be used to view the activation contrast in more detail, by default matplot sets min and max to the actual min and max values of the output
# plt_num: used to plot out multiple different weight feature map sets on the same block, just extend the plt number for each new feature map entry

def outputFeatureMap(image_input, tf_activation, activation_min=-1, activation_max=-1 ,plt_num=1):
    # Here make sure to preprocess your image_input in a way your network expects
    # with size, normalization, ect if needed
    # image_input =
    # Note: x should be the same name as your network's tensorflow data placeholder variable
    # If you get an error tf_activation is not defined it may be having trouble accessing the variable from inside a function
    activation = tf_activation.eval(session=sess,feed_dict={x : image_input})
    featuremaps = activation.shape[3]
    plt.figure(plt_num, figsize=(15,15))
    for featuremap in range(featuremaps):
        plt.subplot(6,8, featuremap+1) # sets the number of feature maps to show on each row and column
        plt.title('FeatureMap ' + str(featuremap)) # displays the feature map number
        if activation_min != -1 & activation_max != -1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin =activation_min, vmax=activation_max, cmap="gray")
        elif activation_max != -1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmax=activation_max, cmap="gray")
        elif activation_min !=-1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin=activation_min, cmap="gray")
        else:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", cmap="gray")